CUSPIDAL DISCRETE SERIES 
FOR SEMISIMPLE SYMMETRIC SPACES 



NILS BYRIAL ANDERSEN, MOGENS FLENSTED- JENSEN 
AND HENRIK SCHLICHTKRULL 

Abstract. We propose a notion of cusp forms on semisimple symmetric spaces. 
We then study the real hyperbohc spaces in detail, and show that there exists 
both cuspidal and non-cuspidal discrete series. In particular, we show that all 
the spherical discrete series are non-cuspidal. 



1. Introduction 

The main purpose of this paper is to initiate a generahzation of Harish-Chandra's 
notion of cusp forms for a real semisimple Lie group G to the more general case of 
a semisimple symmetric space G/H. In Harish-Chandra's work on the Planchcrel 
formula for G the fact that all discrete series are cuspidal plays an important role. 
However, in the established generalizations to G/H (see [11], [16], |9], [7], |8]), 
cuspidality plays no role and, in fact, is not defined at all. 

We propose a notion of cuspidal discrete series for semisimple symmetric spaces 
G/H in general, and we show by explicit calculations on the real hyperbolic spaces 
SO(p, g -f \)e/'S>0{p,q)e that the notion is meaningful in that case. The notion 
agrees with the standard one of Harish-Chandra for the discrete series of G, but in 
contrast to the situation for G, it is not true in general that all discrete series are 
cuspidal. Our main result determines exactly which discrete series representations 
for SO{p,q + l)e/SO(p, q)e are cuspidal. If p > g — 1, all discrete series represen- 
tations are cuspidal, but if p < q — 1, there is a non-empty and finite family of 
non-cuspidal discrete series. 

The notion of cuspidality relates to integral geometry on the symmetric space by 
using integration over a certain unipotent subgroup N* C G. The definition of N* 
is given in Section |2j The map / i~> J^, f{-nH) dn, which maps functions on G/H 
to functions on G/N* , is a kind of Radon transform for G/H. A discrete series 
subspace of L'^{G/H) is said to be cuspidal if it is annihilated by this transform 
(assuming the convergence of the integral on an appropriate dense subspace of 
L^{G/H)). In the group case G ~ G x G/G, we have N* = iV x {e}, where N 
corresponds to a minimal parabolic subgroup of G, and thus the Radon transform 
of a function / on G is the function f{xny~^) dn on G x G. It follows that the 
annihilation by this transform agrees with Harish-Chandra's cuspidality condition 
for the minimal parabolic subgroup. 

It is clear that certain discrete series for G/H, which are spherical (that is, 
they contain the trivial A"-type), cannot be cuspidal since they contain functions 
taking only positive values. Obviously a positive function cannot be annihilated by 
integration over any subgroup N* C G. The present investigation shows that for 
the hyperbolic spaces all spherical discrete series are non-cuspidal, but also that in 
general there exist non-cuspidal, non-spherical discrete series. The non-spherical 
non-cuspidal discrete series are given by odd functions on the real hyperbolic space. 
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which means that they do not descend to functions on the projective hyperbohc 
space. 

The first section of the paper describes in more detail the suggested program for 
general symmetric spaces and motivates our study of the hyperbolic spaces. The 
hyperbolic spaces are treated in the following sections. Apart from the motivation, 
this treatment is to a large extend independent of the general theory. The defini- 
tion of N* and the generalized notion of cuspidality was introduced by the second 
author in lectures at Oberwolfach (2001). The results presented in this paper were 
announced in jl8j . 



We first recall from Harish-Chandra [13] the notion of cuspidality for G and 
its relation to the Plancherel decomposition. Let G be a connected semisimple 
real Lie group with finite center (or more generally, reductive of Harish-Chandra's 
class), and let if C G be a maximal compact subgroup with corresponding Cartan 
involution 9. Let q = t (B p denote the corresponding decomposition of the Lie 
algebra, and let a C p be a maximal abelian subspace. 

Let C(G) denote the Schwartz space for G, which is dense in L^(G). By definition, 
a cusp form on G is a function / G C(G) such that 



for all parabolic subgroups P = MAN C G, and all x,y £ G (the integral converges 
absolutely for all / € C(G)). Let L^g(G) denote the sum of all the discrete series 
representations in L^{G). It is both left and right invariant, and the intersection 
Cds(G) = Ljg(G) n C(G) is a dense subspace. 

Theorem 2.1 (Harish-Chandra). Cds(G) is exactly the space of cusp forms. It is 
non-zero if and only if G and K have equal rank. 

The Plancherel decomposition splits L^{G) into a finite sum of series, each of 
which is related to a particular cuspidal parabolic subgroup P = MAN (that is, 
rankM = rankM n K). The splitting can be accomplished as follows. 

Let f)i, . . . , [}r be a complete (up to conjugation) set of 6'-stable Cartan subalge- 
bras in g, and let Oj = t}i Hp for i = 1, . . . , r. For each i = 1, . . . , r, let Pi be a 
parabolic subgroup with Langlands decomposition MiAiNi such that A; = expo^. 
We can arrange that ai = a, then Pi is a minimal parabolic subgroup. 

We now define Ci{G) C C(G) as the set of functions / S C(G) for which 

• / is orthogonal to all h G C(G) with Jjy h{xny) dn = 0, for all x,y G G. 

• Jj^ f{xny) = 0, for all x,y £ G, for all cuspidal parabolic subgroups some 
conjugate of which is properly contained in Pi. 

In particular, for i ~ 1 the second condition is vacuous, and Ci(G), which is 
called the most- continuous series, is just the orthocomplement of space of functions 
annihilated by all integrals g(xny) dn. On the other hand, for rankG = rank iiT, 
we can arrange that a^- = {0} and Nj. = {e}. Then for i = r the first condition is 
vacuous, and Cr{G) is the space Cds(G) of cusp forms. 

Theorem 2.2 (Harish-Chandra). The following is an orthogonal direct sum 



In Harish-Chandra's Plancherel decomposition, each piece Ci{G) (or its closure 
in L'^{G)) is further decomposed into generalized principal series representations 
induced from Pi. 



2. A GENERAL NOTION OF CUSPIDALITY 



(1) 




C(G) = ®UC,(G). 
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Let now G / H be a semisimple symmetric space, that is, the homogeneous space 
of G with a subgroup H satisfying G1 C H C G"^, where a: G — > G is an involution, 
G'^ the group of its fixed points, and Gg the identity component of this group. The 
problem of obtaining the Plancherel decomposition for L^(G/i?) has been solved 
(see the references cited in the introduction). In general terms the outcome is similar 
to what was described above for L^(G). In particular, discrete series occur if and 
only if G/H and K/K n H have equal rank. 

One can also define a Schwartz space C{G/H) for G/H, and again (see [7], 
Theorem 23.1) there is a finite decomposition 

(2) C{G/H)^®,C,{G/H) 

where each piece decomposes as a direct integral of representations induced from 
a particular parabolic subgroup. However, the pieces in this decomposition are 
defined representation theoretically. The motivation behind this paper was to study 
the following problem. 

Question: Is there a description of the Ci{G/H) through integrals over subgroups 
N (or N/N n H), similar to that for G? In particular, can the discrete series be 
characterized through some reasonable definition of cusp forms? 

Recall that a mmima/ cr^-stable parabolic subgroup PCT-min is obtained as follows. 
Let g = [) © q be the decomposition according to a. We may assume that cr 
and 9 commute, and can arrange that a is cr-invariant and that Oq := a fl q is 
maximal abelian in p H q. The set S of non-zero weights of Oq in g is a root system, 
and Pa-min = M AqN is determined from a choice E+ of positive roots. Here 
Aq = expttq, and MAq is its ccntralizer. The most-continuous series for G/H, 
which is a basic summand in ([2]) , is induced from a parabolic subgroup of this form 
(see [5]). More generally, the representations in Ci{G/H) are induced from a (not 
necessarily minimal) crS-stable parabolic subgroup Pi (see [8], Theorem 10.9). 

It would be tempting to apply the unipotcnt radicals Ni of the Pi in a definition 
of cusp forms on G/H: 

(3) / fignH) dn = {geG,P,^G). 

JNi 

In the group case, where G is considered as a symmetric space for G x G, the aO- 
stable parabolic subgroups of G x G arc of the form P x 0{P), where P C G is 
parabolic, and thus the integral (jSj becomes an integral over both N and d{N). 
Hence ([3]) differs substantially from Harish-Chandra's definition ([T|) in this case. 
Furthermore, although ([3]) docs converge in the group case (see [19], Lemma 15.8.1), 
this is not the case for general symmetric spaces. An example is provided below in 
Lemma WA\ (see however [T5] for the special case of L^-discrete series for G/H). 

Based on these observations one is lead to look for integrals over different sub- 
groups, and the following approach was suggested by the second author. We first fix 
a system of positive roots for a in g, such that S+ consists of the non-zero restric- 
tions to aq. Since was already given, this only amounts to a choice of positive 
roots for the root system of pure Uh-roots, that is, the roots of a which vanish on 
Uq. On Oq an ordering is determined by S+. On wc choose an ordering which 
is compatible with the positive pure roots. More precisely, these ordcrings can be 
attained by choosing elements Xq e aq and Xh G such that a{Xq) > for all 
a G S^; and l3(Xh) > for all positive pure a/i-roots /S. Furthermore, we request of 
Xh that a{Xfi) ^ for all roots of a with non-zero a/i-restriction. Then Xq and Xh 
determine the corresponding notions of positivity for elements in a* and . Notice 
that the notion which results from the choice of Xh is in general not unique. 

We now define the following subspaces (in fact subalgebras) of the Lie algebra n 
of iV: 
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n+ = J2f3 S*^! where /? is a root with I3\ag > and /3\a^ > 0. 
n_ = (T6'(n+) = J2p O*^; where /3 is a root with f3\ag > and /3|ah < 0. 
no = J2i3 ' where /3 is a root with /3|o^ > and /3\a,^ = 0. 
Then n = n+ © no © n_ , and 

(4) n* := n+ © no = ^ fl^ 

/^U,>0,/3|a^>0 



is a subalgebra. Let n** = n_ such that n = n* © n**. Similarly, let A^* = exp(n*) 
and N** = exp(n**), then N ~ N*N**. Notiee that N* intersects trivially with H, 
since this is the case already for N. 

The suggestion is to replace N by N* in ^ and consider integrals of the form 

(5) / fign*H)dn* 

JN* 

in a possible definition of cusp forms on G/ H . It is easily seen that in the group 
case, the integrals (O amount exactly to those in Harish-Chandra's original integral 
([1]) for minimal parabolic subgroups. 

It is useful also to view N* as a quotient of the nilpotent part of a particular 
minimal parabolic subgroup Pi of G. For this purpose we define 

(6) ni = n+ ©no ©6'(n_) ©^fl^, 

P 

with the final summation over the positive pure o/i-roots. It follows from the max- 
imality of o^, that the sum in ([6]) is contained in f). Using this and the fact 

that cr: n_|- — )■ 6'(n_) is bijectivc, one concludes easily for the Lie algebra 

ni = n* © (ni n [)). 

Let Ni = exp(ni), then the following holds similarly. 

Lemma 2.3. The mapping (ni,7i2) i— > nin^ is a diffeomorphism of N* x (A^i ni/) 
onto Ni. 

Proof. The map (711,712) 1— >■ ni7i2 is clearly injective. By }14j . Lemma VI 5.2, it 
is a diffeomorphism onto an open subset N*{Ni n H) of Ni containing Nq. Let 
at ~ exp{tXh). Then limt^oo(a_t7iiat) £ A^o for all ni € TVi, whence a-tuiat S 
N*{Ni n H) for t sufficiently large. Since both TV* and (iVi n H) are normalized by 
at, it follows that 711 e N*{Ni D H). Therefore Ni = N*{Ni n H), and the resuh 
follows. □ 

We thus have 

N* ~ Ni/Ni n H. 

The construction of N* is motivated by the observation that it is the smallest 
subgroup of N which can be realized as a quotient of this form for some minimal 
parabolic subgroup of G containing A. 

For a function / on G/H, we define its Radon transform Rf by 

(7) Rf{g):^ I f{gn*H)dn*=[ f{gniH)dni, (geG), 

Jn' JNi/NinH 

provided the integral converges. 

Let C{G / H) denote the Schwartz space for G/H (see [2]). At the time of the 
present research we expected that ([7|) would converge for all / G C{G/H). However, 
recently it has been suggested by van den Ban and Kuit (see [3], [4]) that the 
definition of N* needs to be refined. More precisely, one needs in addition that 
the element is so chosen, that the roots of n.]- have non-negative inner product 
with the sum of the positive pure a/j-roots (this can always be attained). For the 
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hyperbolic spaces investigated in the present paper, this extra condition is always 
fulfilled. 

Assuming that Rf is well defined for / S C{G/H), one can define the cuspidal 
discrete series for G/H to consist of those discrete series for which the corresponding 
functions in C{G/H) are annihilated by R. 

We shall need a result about the relation between R and invariant differential 
operators on G/H. We let PCT-min = MAqN be as above. Since 

= n® (mn q) ® ttq ® [}, 

we can define a map 

fi : n{G/H) B{M/M n H) (g) B{Ag) 
by fi{D) = T{Do), where 

(8) u-uo enU{g) +U{g)l), 

and u G U{g)^ , uq G U{m)^"^^ (S)U(aq) arc elements that represent D and Dq, and 
where T{Dq) — a~'^Do o a'' (see for example [2], p. 341). The map is independent 
of the choice of positive system for a^. 

Let m„c be the ideal in m generated by m n p. It follows from maximality of Og 
that Tn„c C (). The complementary ideal rric is contained in i and centralizes a. Let 
Mc C M he the corresponding analytic subgroup. Using the decomposition 

(9) m = trie ffi m„c, 
and the fact that m„c C f), we see that 

D{M/M n ~ B{Mc/Mc n H). 

Therefore, we may as well regard /i as a map 

^ : niG/H) ^ BiMc/Mc n H) (g> D{Ag). 

We denote by p, p* , p** , pi E a* half the sum of the roots of n, n*, n**, and ni 
respectively (with multiplicities). Then p = p* + p** and 

Plla, = (P* - P**)\a,- 

Let / be a smooth function on G/H, such that the defining integral of Rf allows 
the application of right derivatives by all elements from 

Lemma 2.4. Let 

(10) Afima) := aP'Rfima), 
for m G Mc and a E Ag. Then 

AiDf) = ti{D)Af, 

for D e B{G/H). 

Proof. Notice first that since McAq centralizes a, it preserves N* in the adjoint 
action. Moreover, the pull-back of the invariant measure dn* by the action of 
ma € McAq is a~'^P' dn*. It follows that 

(11) Af{ma)=aP' I f {man* H) dn* = a- p j f{n*maH)dn*. 

Let u and uq be as above, and note that as remarked above we may assume 
Uq G U{mc + cig)- We shall prove that ([5]) implies 

(12) u-uo en*W(0)+W(0)f), 

from which the desired property of A{Df)(ma) then follows by application of u 
from the right in the last expression in pT|) . 

By Poincarc-Birkhoff-Witt u — uq, modulo U{g)t), is a sum of terms of the form 
Xi ■ ■ ■ XkYi ■ ■ - Yi where Xi, . . . , Xk are root vectors in n, say for roots ai, . . . , a^, 
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and Yi, . . . , Y/ belong to (m fl q) + a^. Wc arrange that the basis elements Xi for 
n are ordered such that roots of n* come first. Since u — uq commutes with a^i, 
it follows from the uniqueness of the expression that ai + ■ ■ ■ + ak vanishes on ah 
for all non-zero terms. As the roots of n** are strictly negative on some element in 
a/i, it follows that in each non-zero contribution at least one root vector Xi must 
belong to n*. □ 

Notice that if / is an eigenfunction of the Laplace operator L on G/H, then it 
follows from from Lemma [2.4[ that Af is an eigenfunction for /x(L) on McA with 
the same eigenvalue. The operator fi{L) is explicitly determined in [5], Lemma 5.3. 
In particular, if Ale C H, it follows that if Lf — cf, then 

(13) {LA-p'){Af) = cAf, 

on A. Here La is the (Euclidean) Laplace operator on A, normalized compatibly 
with the normalization of L. When we define L to be the image of the Casimir 
element in IA{q), this means that correspondingly La is the image of the Casimir 
element mU{a). 



3. Notation and definitions for real hyperbolic spaces 

In this and the following sections G = SO{p, q + l)e denotes the identity com- 
ponent of SO{p,q + 1) and H = SO{p,q)e the identity component of SO(p, g), 
embedded in the upper left corner of G as the stabihzer of xq = ep+q+i G M''+'?+^. 
Throughout we assume p,q > 1. Then G/H is a non-Riemannian symmetric space. 
The corresponding involution ct of G is obtained from conjugation by the diagonal 
matrix (1, . . . , 1, —1). The fixed point group G°" has two components, H and He, 
where c S G is the diagonal matrix (!,...,!, —1, —1). 

It is known that G/H is simply connected except for q = 1, where G/H has an 
infinite-folded covering. This means that for q = 1 we can get a somewhat more 
general result by going to coverings. 

The map G 9 g i— > gxo induces an identification of G/H with the real hyperbolic 
space X = Xp^q, defined by the equation 



= -1 



m IK'' ' ' Likewise G/G"^ is identified with the projective real hyperbolic space 
PX, in which antipodal points x and —x are identified. 

The group K = Ki x K2 = SO(p) x SO{q + 1) C G is a maximal compact 
subgroup, of which the corresponding Cartan involution will be denoted 9. We 
define one-parameter abelian subgroups A = {at} C G and T = {ke} C K2 by 



at 










and 



/ Ip 








' \ 





cos 9 





sin 9 













V 


— sin 9 





cos 6* J 



where Ij denotes the identity matrix of size j. Then 

(14) kgatXo = (sinht, 0, . . . , 0; sin 6* cosh t, 0, . . . , 0, cos 9 coshi). 

The semicolon, which will be used again later, indicates the separation of the first p 
from the last q + 1 coordinates. The generalized Cartan decomposition G = KAH 
holds and gives rise to the use of polar coordinates on X : 

K xW3 (fc, t) ^ katH e X. 
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In this case wc have in addition that Ki C H and K2 = (K2 H H)T{K2 fl H), where 
K2 n H = SO{q) centrahzcs A. Hence 

(15) K = {Kr\ H)T{K2 n H) and G = (A' n H)TAH. 

In particular, wc shall deal with functions / on G/ H which are K n i/-invariant 
from the left. It follows that such a function is uniquely determined by the values 
f{keatH) for {9,t) G [0, 27r] x R. Notice, that the antipodal point corresponding to 
kgatH is kg^^a-tH ■ 

The i^-types with a. K f] _ff-fixcd vector arc generated by the K n i7-bi-invariant 
zonal spherical functions on K . In the present case the zonal spherical functions on 
S0(9 + 1)/S0(g) are given by 

ke i?^(cos6'), 

when g > 1, where is a particular Gegenbauer polynomial of degree ^ G (i.e. 
/X G Z and /x > 0). We normalize these by i?p(l) = 1, and note that in particular 
i?i(cos0) = cosO. When g = 1, we also allow the integer /i to be negative, and 
replace R^{cos9) by 

kg^e''''^. 

It follows that a function / on G/H which is A'-finite of irreducible type ^ and 
K n iJ- invariant, must be of the form 

(16) f{keat)^R^{cose)f{at), respectively f{keat)=e'^''>f{at). 

In the study of discrete series on scmisimplc symmetric spaces G / H , one often 
needs the following general fact: 

Lemma 3.1. Let V he an irreducible, non-zero closed invariant subspace of L^{G/ H) 
or G°°{G/H). Then V contains a function f with the properties: 

(a) /(e) = 1, 

(b) / is an eigenfunction for the Casimir operator on G / H , 

(c) / is K n H -invariant and of some irreducible K-type /x. 

Since V is irreducible, it is generated by this element /. 

The number /x is related to the highest weight of the AT- type as follows. Let T £t 
be the infinitesimal generator of kg, then i/x is the value of the highest weight on T 
(with a suitable choice of positive restricted roots for t) . 

3.1. Parametrization of discrete series. We define 

(17) p^l{p + q-l) and p,^\{q~l), 
and for A > 

^\ = \ + p - 2pc. 

We first consider the case g > 1. It is known (see for example [11], Section 8) 
that the discrete series for the hyperbolic space G/H is parametrized by the set 
of positive numbers A such that £ 2- The representations that arise from the 
construction for general semisimple symmetric spaces in |11| are exactly those for 
which px > 0. The remaining discrete series (called 'exceptional' in [H]), corre- 
spond to those (finitely many) parameters A > for which /x^ < 0. The exceptional 
parameters exist if and only if g > p + 3. The discrete series representation with 
parameter A descends to the projective space FX if and only if px is even. For gen- 
eral semisimple symmetric spaces, the full discrete series (including the exceptional 
ones) is described in [T6] . 

For q = 1, where pc = 0, we parametrize the discrete series by A G M\{0} such 
that |A| + p G Z. In this case we have /x^ = A + p > 1 for A > 0, whereas for A < 
we define 

/XA = A - p < -1. 
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There are no exceptional discrete series. We note that for q = I every A 7^ defines 
a relative discrete series for the infinite covering space of G/H . 

We return to the general situation g > 1, and describe the discrete series which 
are spherical, according to [11], Theorem 8.2. Spherical discrete series exist if and 
only if (7 > p + 1, and in this case, the representation with parameter A is spherical 
if and only '\i ^\ <Q and even. 

The discrete series parameter A is related to the eigenvalue of the Laplace- 
Beltrami operator A of G/H on the corresponding representation space in Lp'{G/ H). 
More precisely, we have A/ = (A^ — p^)f , for functions / in this space (with suit- 
able normalization of A), and we can explicitly describe the discrete series by a 
generating function of the form as follows. 

Proposition 3.2. Let A G M \ {0} he a discrete series parameter (if q > \, this 
implies in particular that \> Q). The corresponding discrete series representation 
T\ has a K n H -invariant generating function of the following form 

(i) For q > I and > 0, 

i^\{keat) = i?^Jcos6i) (cosh i)-^"''. 

For q = 1 and jix G Z, or for all A for the relative discrete series for the 
universal covering, 

iPxikgat) = e'^^" (cosht)-!^!-". 

(ii) For q > p + S, ~ — n < and 

(I) n — 2m even, 

Ukeat) = PA(cosh2t)(cosht)-^-''-2™^ 

(II) n = 2m - 1 odd, 

^x{keat) = cos 61 Px{cosh^ t){cosht)-^-P-^"\ 

where in each case P\ is a polynomial of degree m. 

Proof. The expressions for the generating functions can be derived from known 
explicit formulas with hypergeometric functions for the i^- finite functions on G/H, 
see [l7l, p. 1864, or ^0\, p. 403. However, we prefer to give an alternative proof 
which relates more directly to general theory. 

For (i) wc refer to [TT], formula (8.11), and for (ii) we refer to [T^], where explicit 
expressions arc determined for the generating functions of the exceptional discrete 
series. It follows from [12], Theorem 5.1, that the following function generates the 
discrete series with parameter A: 

(I) n = 2m even, 

£,x{keat) = (/i„,m(sinh^ t), 

(II) n = 2m - 1 odd, 

^x{kgat) — cos6'cosht 0„_m(sinh^ i). 
Here 4'n,m is the function on defined by 

(18) 0n,™(s2) = [c^"(l + a:')"-''1|.=(s,o 0), 

where a; G K^, and where uj denotes the Laplace operator on W. 

Note that ^\ differs from the function constructed in [T^ by being K n H- 
invariant. In the notation of |12j . the K n 7?- invariant generating function is 

In order to prove the proposition it now suffices to show for each relevant pair 
(n, m) that there exists a polynomial P of degree m such that 

(19) </.„.™(s2) = P(s2)(l + s2)"-2™-Pe, (sGM) 
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for all s G M. The expression is derived from ([T5|) by successive use of the 
following lemma. Note that n — pc = —X — p + Pc < □ 

Lemma 3.3. Let Q be a polynomial of degree d and let i/ G M. Then there exists a 
polynomial P of degree < d + I such that 

w(Q(a;2)(l + x^Y) = P{x^){l + x^)"-^, {x G W). 

If + d ^ and u + d ^ then deg P = + 1 

Proof. The existence of the polynomial P is an easy computation. The statement 
about its degree follows from the fact that 

(5(x-^)(l + x'^)" = {x'^Y+'^ + lower order terms, 

since w(.t^) = fi{^ + p - 2)a;^"^ for all /i G M. □ 

4. A UNIPOTENT SUBGROUP 

Define 



cxp{Zu.v) G G, with Zu,' 



where u G M.p~^ and v gM.'^ are considered as rows, and u*, v* are the corresponding 
columns. If Y denotes the infinitesimal generator of at , then [Y, Z^^v] = Z^^v for all 
M, V. The matrices Z.^^v span a commutative subalgebra n of g, and the subgroup 
N — exp(n) is the unipotent radical of a minimal cr^-stable parabolic subgroup P 
in G. In particular, we note that N D H is trivial. 
Easy calculations show 

(20) nu,vXQ = - v^),u; -v, 1 + 1(^2 - v^)), 
and 

(21) agUu^v^o — (sinh s + ie'*(u^ — w^), u; —v, coshs + ^e/^u^ — v'^)), 
for all s G M, where u'^ = u ■ u and = w • u as usual. 

Lemma 4.1. Assume p > I and p + q > 3. Then there exists a non-negative K- 
invariant Schwartz function f G C(G/H) for which the integral Jj^ f{n) dn diverges. 

Assume in addition q > p + I. Then the integral diverges for the K -invariant 
discrete series function f ~ ip\ where A = ^{q — p — 1) (see Provosition \3.S\( i)). 

Proof Let f{katH) (cosh t) with > 0, then / G C{G/H) (see the remark 
after Theorem [SH]) . Using (HH) and ([20]), we find 





u 


V 


° 1 










u 












V 


u 


V 


/ 



N 



f{n)dn^ / / f{nu,v)dudv 



oo />oo 



{V^ + (1 + i(w2 - t;2))2)-^(p+-) 

L 2 

(y2 + (1 + _ J/2))2)-i(p+.)^p-2yg-l ^^^^^ 

"'0 2 

In particular, for 1 < j/ < a; < y + 1, we have < x'^ — y'^ = {x — y){x y) < 2y + 1, 
and hence 

y' + il + kx'-y')f<10y'. 
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Hence 

/ f{n)dn> / / {10y'^yi^''+''^xP-'^y'^-^ dxdy 

Jn Jl Jy 

/oo 

with C > 0. This integral diverges when ly < ^{p + q ~ 3). 

If (7 > p+1 and A = ^{q — p—1), then the function / defined above with = A is 
exactly i/'a- The integral diverges since in this case A = ^{q—p—1) < ^{p + q — 3). □ 

If p + g < 3, it is likely that the A^-intcgral converges for all Schwartz functions, 
but we shall not consider this question here. 

Motivated by Section [51 we now define the following subgroup of N. Let 

u = (ui, . . . , tip_i) e R^"^ and v ^ {vq, . . . ,vi) €W . 

It is convenient to number the entries of v from right to left as indicated. It is not 
difficult to verify that the following agrees with (U). 

Definition 4.2. Let N* C N be the max(p — 1, q)- dimensional subgroup 

N* = {nu,v I u G RP-^,v e R«, lij Vj for j = 1, . . . , I}, 
where I = niin(p — l,q). 

We want to integrate K n iJ-invariant functions on G/H over sets of the form 
ttsN*, where s £ M, with respect to Haar measure of N*. For this purpose we shall 
need the following, which is easily deduced from (|14p and ([^1]). 

The relation 

{K n H)keatH = (K n H)asnu^vH 

implies 

cosh^t = + (coshs + ie-'(u^ - u^))^ 

and 

coaO = (coshs + ^e/{u'^ — w^))/cosht 

for all 6, t, s, u and v. 

When p = 1, the value of t, including its sign, can also be determined by 

sinht = sinhs + ^e^{—v'^), 

whereas if p > 1 the sign is redundant since by ([M]) the double cosets {Kr\H)k0atH 
and [K n H)kga-tH are identical. We assume in this case that t>Q. 

With these relations between (s, m, v) and (6*, t), we have /{asUu^vH) = f{kgatH) 
for K n _ff-invariant functions / on G/H. 

We assume now e A^*. Then the expression u^—w^ simplifies. Wc distinguish 
the two cases: 

A. p > q. Then u = {vi, . . . , Vq,u'), where u' e Rp^^^"?. We put x = \\u'\\ and 
y = \\v\\, and obtain 

(22) cosh^ t = y^ + (coshs + ^e/x^)^. 
and 

(23) cos 6* = (coshs + ie'^a;^)/ cosh t. 

For the integration over N* we use polar coordinates for u' and v with 

(24) a^p-2-q, /3 = q-l. 

We conclude that the measure on N* can be normalized such that for a K n H- 
invariant function, 

f /"OO /"OO 

(25) / f{asn*H)dn*^ / f{kgatH) x'^y^ dx dy, 

JN" Jo Jo 
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where t = t{s, x,y) > is determined by and 6 ~ 6{s, x, y) by 

Note that in the degenerate case p — 1 = g, we have u' = 0. Hence a; = in 
and the right hand side of ((25|) has to be interpreted without the integration over 

X. 

B. q> p. Then v — (v' , Up-i, . . . , ui), where v' e E'J^p+^. We put x = \\u\\ and 
y = \\v'\\, and obtain 

(26) cosh^ t = x^ + y'^ + (coshs - ^e^'y^ f. 
and 

(27) cos 6* = (coshs - ie^y^)/ cosh i. 
We use polar coordinates for u and v' with 

(28) I3^q~p. 

Then ^ holds with t = t{s,x,y) > determined by (EH) and 6 = 9{s,x,y) by 
(HZl). 

In the degenerate case p = 1, the sign of t is determined by 

sinht = sinhs — ^e'^{y'^). 

Note also that in this case w = 0, so that x = in and again (pS]) is interpreted 
without integration over x. 

To summarize, let us define for s,x,y £ M 

(29) efs X + + (^^'^^ 

Then in particular for a A'-invariant function on G/H^ we can write 

(30) / f{asn*H)dn*= / / F(e(s, x, 2/))x"/ dx dy, 
Jat* Jo Jo 

where F is the function F(cosh^i) = f{atH) on ]R_|_, and a and /? are given by 



either ([24| or (|28l) . 

We have normalized the measure on N* such that this integral equation is valid 
without a constant. 

5. Main results for real hyperbolic spaces 
For functions on G/H we define, assuming convergence, 

Rf{9)= I f{gn*H)dn\ (geG), 

in accordance with the definition of the Radon transform in Section [2l 

We shall be particularly interested in the values of Rf on the elements Us- For 
simplicity we write 



Rf{s) = Rfias) = / f{asn*H)dn*, {s 



/at. 

For A'- invariant functions this integral is explicitly expressed in pop . 
Referring to Lemma |2.4| we find 

^f^P-?-!) ifp>9 
\\{q-p^\) ifp<9, 

and recaU that Aj{a) = aP^Rf{a). It follows from ^ that if Lf = (A^ - p^)f, 
then {d/ds)^Af = X^Af, and hence Rf is a linear combination 

(31) Rfis) = Cie'--P'+^^'' + C2e^-P^-^^\ 
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Theorem 5.1. Let f be a continuous function on G/H, and assume there exists a 
constant C > 0, such that 

(32) \f{kat)\ < C (coshi)-" (1 + log(cosh^))-^ 

for allt eR, k e K. 

(i) Convergence. The defining integral of Rf{s) converges absolutely for all 

seR. 

(ii) Compact support. If f{kat) = for all k G K and \t\ >tQ>0, then 

Rfis) = 0, 

A: for all \s\ > "i-f P > Q, 
B: for all s < —to if P ^ Q- 

(iii) Decay. Let N > 2, and put 

(33) ||/|U= sup (coshO''(l + log(cosht))^|/(fcat)|. 

teVi.keK 

Assume ||/||jv < oo. There exists a constant Cn > (independent of f), 
such that 

eP-^\Rf{s)\<CMN{l + \s\)-^''-^\ 

a: for all s €M. if p > q, 
b: for all s < if p < q. 

(iv) Limits. Assume that p < q. The function e'^Rf{s) is bounded on K"*". /// 
is K -invariant, then the limit lims^co e.^Rf{s) exists, and if in addition f 
is positive and not identically zero, then this limit is positive. 

Note the difference between the conditions for A, B versus a, b. 

We also remark that a Schwartz function / e C{G/H) by definition, see [T], 
Definition 2.f, satisfies the growth conditions ||Z)/|[Ar < oo, for aU D S 1D){G/H) 
and all N eN, with || • H^v defined by In particular, / G C{G/H) satisfies (|32|) . 

Theorem 5.2. Let X be a discrete series parameter, and let f be the generating 
function of Proposition \3.S\ 

(1) // A > and fix > 0, then Rf = 0. Likewise if \ < and fix < 0. 

(2) //A > and fix < 0, then Rf{s) = Ce^-P^+^> {s e M), for some C ^ 0. 

Notice that the second statement in (1) is only relevant when q = 1. In this case 
(2) never occurs, and we always have Rf = 0. This is also the case for all relative 
discrete series parameters (see Subsection 13. 

In conclusion, we have the following characterization of the cuspidal and non- 
cuspidal discrete series. 

Theorem 5.3. Let X ^ be a discrete series parameter. 

(1) Let q > 1. Then the discrete series representation Tx is cuspidal if and only 
if fix > 0. 

(2) Let q < p + I. Then all discrete series are cuspidal (if q — 1, then all the 
relative discrete series are also cuspidal). 

(3) All spherical discrete series for G/H are non-cuspidal. These representa- 
tions exist if and only if q > p -\- 1. 

(4) There exist non-spherical non-cuspidal discrete series if and only if q > p+3. 
These representations do not descend to discrete series of the real projective 
hyperbolic space. 

This follows easily from Theorem 15.21 and the description of the discrete series in 
Subsection 13.11 
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(34) e(., X, y) = + ie^^(y2 - i f + '^v' + \ + \e~'' , 



6. Proofs 

The proofs are based on the following two lemmas. 
Lemma 6.1. For each s G M, there exist numbers a,b > such that 

Ois,x,y)>hl^^l^ ^ 

I X + ay +0 ij q > P, 

for all x,y eW. If p > q or s < 0, then the numbers a, b can be chosen as follows 

a — ^ ~ cosh^ s. 

Proof. The statements for p > q are straightforward from (|29p. Assume p < q. 
From ((29| we obtain 

e(s, X, y) + ie^^'y'* + i(l - e^')y'^ + cosh^ s. 

The statements for s < follow. By rewriting 

(34) e{s,x,y)^2 

we finally see that for s > 

e(.,x,y) >x' + \{y' If ^V^\ = + -/ + i- ° 
Lemma 6.2. Lei a, 6, c, d and 7 fee > 0, i/^en 

/ / (l+x" + j/'')-'^a;"-^/-^dxdj/<oo, 
Jo "'0 

c 

- + - < 7- 
a 

Furthermore 

coo roc 

/ {l + x" + y^)~'^x''-^y'^-^ {I + \og{l + x'' + y^))-^ dxdy < oo, 
Jo 

if S > 1 and 

c d 

- + T=1- 

a 

Proof. Easy. □ 

Proof of Theorem 15.11 We may assume that / is /^-invariant, since otherwise 
we can replace it by the continuous function gH M- sup^.^/^ \f{kgH)\. The defining 
integral (pO)) of Rf{s) is bounded by the following integral 

\Rf{s)\< / x''y^Q{s,x,yrp/^{l + \ogQ{s,x,y))-^dxdy, 



with the values of a, (3, and p from 1^^, (pS)) . and p7|) . The convergence of this 
integral is an easy consequence of the preceding lemmas. Note that the logarithmic 
term is needed for the convergence. This proves (i). 
It is seen from Lemma ISTTl that if p > q or s < 0, then 

9(s, X, y) > cosh^ s. 

We can thus conclude that ii f {at) = for all \t\ > to, then the integrand in ([50]) 
vanishes for \s\ >tQ if p > q, and for s < —to ii p < q. This proves (ii). 

We now study the asymptotic behavior of Rf{s) as s — !• ±c«. The following 
arguments have to be adapted slightly in the degenerate cases p = q + 1 and p = I, 
where there is no x-integral. 
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Assume first that p > q. Recall that a = p — 2 — q and [3 = q — 1, and that 

/•oo poo 

(35) Rf{s)^ / F{Q{s,x,y))x''v^dxdy. 



Jo Jo 

From Lemma lG.f I 

I 

&{s, X, y) >y^ + ^e^^a;"' + cosh^ s, 
and by the definition of H/Hat, we have 

(36) \F{e)\ < cwfue-"/^ (1 + log(e))-^. 

We insert this bound in ([35|) . replace Q{s, x, y) by the lower bound, and substitute 
\e^x^ — cosh(s)^^ and y = cosh(s)77, so that 

(37) _^ 1 g2.^4 ^ ^Qgj^2 5 = (1 + ^2 ^ ^4) ^Qgj^2 



Simplifying by the relation i(Q; + f)+/3 + l = p, we finally obtain 

/>oo />oo 

|i?/(s)| <C||/||^e"^("+l> / / (l+7?2+e4)-p/2(i^i^g(.))-iV^^^a^^^^^ 



Jo Jo 

where ([37)) is the omitted argument in the logarithm. The logarithmic term is 
bounded above by a constant times 

(l+log(l + r/2+n)-2(l + |.|))-(^-2). 

With Lemma [6.21 statement (iii)a follows, except for the case p — q. 

We now assume p < q. Then a — p — 2 and (3 = q — p. For s < 0, we use the 
estimate 

0(s, a;, y) > + ^e^^y'^ + cosh^ s, 

from Lemma |6. II Proceeding as before, with the roles of x and y interchanged, we 
obtain (iii)b and the negative direction of the remaining case in (iii)a. 
If instead we use the estimate 

&{s,x,y)>x^ + \e^^{y'-lf, 

which follows from (p4)) . and substitute x ~ ^e^jy^ — we obtain for s > 

\Rfis)\ < Ce-'^\l + sf-^. 

Here C £]0,co] is given by 

/>00 /"OO 

c= {i+er'^'cdd iy2-i|(''-«-i)/2(i+iog(.))-Vd2/, 



where argument in the logarithm is the maximum of 1 and (y^ — 1)^. The integral 
is finite precisely when p = q, and in this case we thus obtain the desired bound in 
the positive direction. This finishes the proof of (iii). 
Assume p < q. Recall that 

e(s, X, y) = a:^ + y^ + (cosh s - ^e^'y'^f. 

Let V = — sinhs + l/2e'^y'^, then y^ = 1 + 2e^-''v — e^^''' and 

e(s,a;,y) = l + x^ +v^. 

With this substitution, we find 



oo /'OO 



(38) Rf{s)^e-' / F{l + x^ + v^){l + 2e-'v-e-^')^^-^'>^^x''dvdx, 

Jo J — sinh s 

with the following upper bound, for s > 0, since /3 > 1: 

poo poc n I ^2 I 2\-p/2 

\Rmi < Ce- I j ^ ^,\t,ilXxhv^)? ^' + 2H)(^-)/V d. dx < +0O. 
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Applying Lcbcsgue's theorem, we get 

lim e'Rfis) = / / F(l + + v^)x'^dvdx 

"-^"^ Jq J-oo 

from which (iv) follows. □ 

Before we proceed, we note that if / satisfies a sharper decay than ([5^ . we can 
improve on the decay of i?/(s) expressed in (iii). 

Lemma 6.3. Let f be a continuous function on G / H , and assume that 

sup (cosh I /(feat) I < OO 
teB,fce-ftr 

for some 7 > 0. Then for each e > there exists a constant C > 0, such that 

eP'" \Rf{s)\ < C{coshs)-^+\ 

a: for all s €R if p > q, 
b: for all s < if p < q. 



Proof. Replace ([36)1 by 

|i^(e)| < ce-^''+'^)/2 

in the proof of (iii) above and proceed as before. □ 

Proof of Theorem 15.21 Let A be a discrete series parameter, and let / be the 
generating function of Proposition 13.21 We know, sec ([SI]) , that s n> Rf{kas) is a 
linear combination of e^~'''-^'^'>^ and e'-^''^^'^^'* for each k & K. 

We shall first establish conclusion (1) in the theorem. Here we use the following 
lemma. The vanishing of Rf will be established by showing that, for some a e M 
and all k G K, e°"^ Rf{kas) decays to as s —00, and that it is bounded in the 
other direction. 

Lemma 6.4. Let : R — 5- M 6e a linear combination of exponential functions with 
real exponents. If (pi^s) is bounded as s ^ 00, and decays to as s ^ —00, then 
= 0. 

Proof. The maximal exponent is < and the minimal exponent is > 0. □ 

For simplicity we assume in what follows that p > I. The arguments for the case 
p ~ I are similar. Notice that if q = 1 and A < 0, then by definition tpx equals 
the complex conjugate of ip-x- Hence Ripx is the complex conjugate of Rtp-x- 
Therefore, to prove Theorem 15. 2f l). we may assume A > 0. 

We first assume > 0. It follows from the expression for f — tpx hi Proposition 
ESKi) that 

\f{kat)\ < C{cosht)-^-P, 

for all A: e if and t € M. 

For p > q, it follows from Theorem 15. II (iii) a that 

e/^'Rf{ka,) 

in both directions s ±cxd, thus Rf{kas) = 0, for all s G M and k E K, and hence 
Rf = 0. 

For p < q, it follows from Lemma [6.31 that e'''i^'*'+^''*i?/(A:as) is bounded for 
s — > — cxD, for all e > 0. Since pi — A = —fix + 1 < 1, we can choose e = ^(1 — pi + A), 
and conclude that 

e'Rf{kas) = e^e'^P^'^+'^^Rfikas) 

for s — )■ —00. On the other hand, from Theorem 15.11 (iv). we infer that e'^Rf{kas) 
is bounded at +00. Thus again Rf = 0. This proves Theorem 15.2( 1). 
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Assume now that fix ~ 0. This only happens when p + 1 < q. From Lemma [ 
we see that eP^'^Rf{s) — > as s — — cio. We know that eP^^Rf{s) is a hnear combi- 
nation of the exponential functions e^^ and e""^*, but because of the limit relation 
at — oo, only the first one can occur. Hence Rf{s) is a multiple of e^^'^^^'^'''^ = e^^ . 
Since / > 0, we conclude from Theorem IS-lT iv) that \m\s^ooC^Rf{s) = C 7^ 0. 
Hence Rf{s) = Ce". 

Assume finally that jix < 0, then p + 3 < q and / = in ProTJOsition I3.2f ii). 
From the expressions in (ii), we infer immediately 

(39) \^x{keat)\<Cicosht)-'-^+p\ 
Furthermore, since Pa has degree m, the limit 

(40) lim (cosh i)^+''G (at) 

exists and is non-zero. 

We first consider the even case n — 2m, where £,x is JC-spherical. We shall need 
the following lemma. 

Lemma 6.5. Assume p < q. Let f be a K -invariant continuous function on G/H , 
and assume that there exists 7 > such that 

sup (cosh i)''+'' I /(at) I < 00, 

t6R 



and such that 

exists and is non-zero. Then 



lim (cosh i)''+T/ (at) 



lim e'^P'-'^''' Rf{s) 

s— > — 00 

exists and is non-zero. 

Proof. Recall that for p < q, we have 

8(s, x, y) = x'^ + (cosh s - ie'^y^)^. 
We make the substitutions 

(41) y^\(^i + e^'{y^ -I)), u^e'x, 
and find 

(42) y"^ = e-'^'{2v + e^'' -1), cosh s ^ ]^e' y"^ = \l - v) , 
so that 

(43) Q{s,x,y) = e-'^'{u^ + v'^) + l. 
Hence Rf[s) equals 

/•oo /"OO 

g_,(„+^+2) / / F(l + £-2^(7.2 + „2))(2t- + e2^-l)('5-l)/2H"di;d?x. 

Note that a + /3-|-2 — q. For s sufficiently large negative we obtain a uniform bound 
e«1i?/(s)| 

<C / [1 + 6-2.(^2 + t;2)]-(7+P)/2(2„ + ^^s _ l)(/3-l)/2^a 

Jo J(l-e2»)/2 

/•oo noo 

Jo Jl/i 

It follows that we can apply Lcbesgue's theorem to the limit 

lim e'^P^-^^" Rf[s) = lim e^-'<-P+''^' RJ{s). 
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Since r'^+PF(r'^) allows a non-zero limit c for r — > oo, it follows that 
for s -> — oo. We conclude that e'^^'^^P^'^^ Rfis) tends to 



oo pOQ 



la Ji/2 

as s — ?► — oo. □ 

Proceeding with the proof of Theorem 15.21 we recall that f = It follows 
from pop and (^0]) that Lemma [6.51 is applicable in the even case n = 2m, and we 
conclude that C := Imis^-oo e^^^^P^^'^ Rf{s) exists and is non-zero. As Rf{s) is 
known to be a combination Cie^'^"''^'''* +C2e^~'^~''i^^, we conclude that Ci = C and 
C2 = 0. Hence in this case 

Rf{s) ^ Ce^^-P'^' = Ce(^^-i)^ 
Finally, we consider the odd case, fix = — n = —2m + 1, where 
/(fceat) = cos 61 PA(cosh^t)(cosht)"^"''"2m^ 

Since / is not ii'- invariant. Lemma 16.51 is not directly applicable. However, we shall 
adapt its proof. It follows from ([27|) that cos cosh t — coshs — ^e~''y^, and hence 

R.f{s)= / x"/ (coshs - -e"?/2)Q(s,a;,y) dec dy, 

where 

As in the proof of Lemma l6.5l we perform the substitutions (|4ip . By application of 
(I42l)-(l43]) we see 



00 pOO 



e(i+^>Rf{s)= / / u"(2t.-Ke2''-l)('3-i)/2(l_t,)Q(s,u,w)di;dit, 

Jo J(l-e2=)/2 

where 

Q{s,u, v) = Px{l + e-^\u^ + v')){\ + e-2-(w2 + ^2^^-i(A+p+i)-m^ 
This gives the domination for s sufficiently large negative 
e(9+i)^|i?/(s)| 

Jo J(l-e2s)/2 

/>00 /'OO 

<Ce.(7+P+l) / / (^2^^2)-(A+p+l)/2^(/3+l)/2^a^^^^^^^_ 
Jo J 1/4 

Again we can apply Lebesgue's theorem and obtain 

/•oo /"OO 

lim e(i-^^)'*i?/(s) = c / / (l-i.)(it^ + w^)-^^+''+^^/^M"(2w-l)('^^i)/2dvdu, 

Jo Jl/2 

with c ^ lim.^oo r-f^+P+iVn.mlr^) ^ 0. 

In order to prove that Rf{s) = Ce^^^"^-***, with C ^ 0, we argue as before. We 
only need to establish that the following integral is non-zero: 



00 poo 



(1 - V){U^ + Wl)/2ya(2„ _ l)(^i-l)/2 ^„ 

/O Jl/2 

We rewrite this by setting u = vx and 2w — 1 = y. Up to a power of 2 we obtain 

/•OO poo 
(44) / (l+x2)-(Wl)/2^a^^ / (l_y)y(/3-l)/2(y+l)-(A+p-o)^y^ 

Jo Jo 
in which the first integral is clearly finite and non-zero. 
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We now use the formula 

r(fc)r(z - k) 



/ y''-\l + y)-'dy = B{kJ-k)^ 
Jo 

valid for < fc < L It follows that for < fc < / - 1 
(l-2/)/-i(l + y)-'dy=(/-2fc-l) 



r(fc)r(/ - fc - 1) 







r(0 

Hence the second integral in pij) is zero if and only if 

(A + p - a) - (/? + 1) - 1 = 0. 

With the current values of a, /?, A and p, we have X + p — a — (5—1 = —n, and hence 
J ^ 0. This finishes the proof of Theorem 15.21 □ 
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